Problem

The Hadoop DFS client auto closes and created clients, when the JVM shuts down. This can happen prior to the connector being closed, which requires these clients to be in open state for deletion of temporary files on close. Note that this error only comes into the picture when the connect worker is stopped/restarted.

Solution

Hadoop has a configuration fs.automatic.close which is by default true. Setting this as false will disable this behaviour. This is safe to do as we are anyways not relying on the auto close feature for closing all the clients. The connector lifecycle has appropriate routines for closing these clients on exit since it needs to rely on the lifecycle during a a connector delete, where the JVM shut down hooks are not executed. The connector should always operate with fs.automatic.close as false.

Does this solution apply anywhere else?

• yes
• no

If yes, where?

Test Strategy

Testing done:

• Unit tests
• Integration tests
• System tests
• Manual tests

Release Plan